§ : On configuration spaces of plane polygons, 

^ '■ sub-Riemannian geometry and periodic orbits 

< \ of outer billiards 

oo _ 

Daniel Genin and Serge Tabachnikov 

^ ■ Department of Mathematics, Penn State University 

Q ■ University Park, PA 16802, USA 

1 Introduction 

. , The classical billiard system describes the motion of a point in a plane domain 

(X5 ! subject to the elastic reflection off the boundary, described by the familiar 

OO ■ law of geometrical optics: the angle of incidence equals the angle of reflection; 

^ ■ see, e.g., [131 Ej for surveys of mathematical bilhards. 

O . For every n > 2, the billiard system inside a circle has a very special 

O 
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property: every point of the circle is the starting point of an n-periodic 
billiard orbit; this orbit is an inscribed regular n-gon.^ Likewise, an ellipse 
enjoys the same property for every n > 3; the orbits are inscribed ra-gons of 

r^ ■ extremal perimeter length.^ A billiard table of constant width also has the 

property that every point on its boundary belongs to a 2-periodic, back-and- 
forth, billiard trajectory, and this is a dynamic characterization of the curves 
r> \ of constant width. The phase space of the billiard ball map is a cylinder, and 

c^ ' the family of n-periodic orbits forms an invariant circle of the billiard ball 

map consisting of n-periodic points. 

How exceptional is this property? More specifically, given n > 3, one 
wants to describe plane billiards such that the billiard ball map has an in- 
variant curve consisting of n-periodic points. The first result in this direc- 

"'^ Similarly, one can consider periodic orbits with other rotation numbers corresponding 
to regular star-shaped polygons. 

^For circles and ellipses, this behavior is due to the complete integrability of the billiard 
ball map. 



tion was obtained by Innami [S]: he constructed a billiard curve, other than 
a circle or an ellipse, whose every point is a vertex of a 3-periodic billiard 
trajectory (these billiard triangles are isosceles in his example).^ 

In a recent paper P, Baryshnikov and Zharnitskii developed a new ele- 
gant approach to this problem. In a nutshell, their innovation was in switch- 
ing focus from billiard curves to the space of plane n-gons. If such an ra-gon, 
Zi, ..., Zn, is a. periodic biUiard trajectory then, due to the law "the angle of 
incidence equals the angle of reflection" , we know the directions of the bil- 
liard curve at each vertex Zi. This determines an n-dimensional distribution 
P in the space of n-gons, called the Birkhoff distribution. One can further 
restrict attention to the hypersurface of n-gons with a fixed perimeter length: 
the Birkhoff distribution is everywhere tangent to these hypersurfaces. 

It turns out that the Birkhoff distribution is non-integrable. A 1-parameter 
family of billiard n-gons zi{t) , . . . , Znit) , < t < 1, is interpreted as a curve 
in the space of n-gons, tangent to V and satisfying the "monodromy" con- 
dition Zi{l) = Zj+i(0), i = 1, . . . ,n. The problem of finding such horizontal 
curves belongs to sub-Riemannian geometry and non-holonomic dynamics. 

The main result of P is that, for every n > 2, the space of billiard 
tables, for which the billiard ball map possesses an invariant curve consisting 
of n-periodic points, is infinite-dimensional (this space also has an infinite 
codimension in the space of all billiard tables). 

Example 1.1 Let us illustrate this approach in the simplest case n = 2, 
that is, curves of constant width (for information on billiards of constant 
width, see 0). 

Consider the space of oriented segments, say, of length 2. A segment ^12:2 
is "allowed" to move in such a way that the velocities of both end-points are 
orthogonal to the segment. Let O = {x,y) be the mid-point of the segment 
and a its direction. Then the space of segments has coordinates {x, y, a) and 

zi = {x — cosa,y — sin a), Z2 = {x + cos a, y + sin a). 

We conclude that the velocity of point O must be orthogonal to the vector 
(cos a, sin a), and this is the definition of the Birkhoff distribution in this 
case. Thus V is the kernel of the 1-form A = cos a dx + sina dy. This 1-form 
is contact: X A dX ^ 0. We have identified our space of segments with the 



^A similar result was obtained by M. Berger, unpublished. 



space of cooriented contact elements in the plane, a fundamental example of 
a contact 3-dimensional manifold (see, e.g., |5)- 

If Zi{t)z2{t) is a curve, tangent to "D and satisfying the monodromy con- 
dition -2i(l) = ^2(0), 2:2(1) = :2i(0), then the respective curve {x(t),y(t),a{t)) 
is Legendrian (i.e., tangent to the contact distribution) and satisfies a;(l) = 
x(0), y{l) = y{0) and a(l) = a(0) + tt. 

The projection of this Legendrian curve on the {x, y)-plane, that is, the 
trajectory of the mid-point O, is a closed smooth curve A with singularities 
(generically, semi-cubical cusps); the total winding of this curve is n. Every 
such curve A uniquely lifts to a Legendrian curve in the space of contact 
elements: the missing a coordinate is recovered as the slope: cot a = —dy/dx. 

In other words, we use A as a guide: place a segment of length 2 so that 
its mid-point O is on the curve A and the segment is orthogonal to A, and 
then slide the segment around A. If certain convexity conditions hold (for 
example, A should not have inflection points) then the end-points zi and Z2 
will describe a curve of constant width. 

For example, if the end-points Zi and Z2 move along a circle then A 
degenerates to its center, and if Zi and Z2 describe the Reuleaux triangle 
then A is made of three 60° arcs of a circle, see figure ^ 




Figure 1: Reuleaux triangle and the curve A 



In this paper we extend the results from ^ to outer (or dual) billiards, a 
dynamical system that is a close relative of the conventional, inner, billiards; 
see jSmnilllj for surveys and j3] for a recent interesting result. 

An outer billiard table is a strictly convex compact plane domain D. Pick 
a point X outside D. There are two support lines from x to D; choose one 



of them, say, the right one from x's view-point, and reflect x in the support 
point. One obtains a new point, y, and the transformation T : x ^-^ y is the 
outer bilhard map, see figure |21 If the boundary of the outer bilhard table 
contains straight segments (for example, is a convex polygon), the map T 
or its inverse is not defined on the lines containing these segments; this still 
leaves a set of full measure on which all iterations of T are defined. We will 
consider only outer billiard tables with strictly convex smooth boundaries. 




Figure 2: Outer billiard map 

Let us mention four fundamental properties of outer billiards, important 
for us. First, the outer billiard map commutes with affine transformations of 
the plane. Secondly, the outer billiard map preserves the standard area form 
in the plane (similarly to the billiard ball map which is an area preserving 
map of a cylinder). Thirdly, the outer billiard map is a twist map of the 
exterior of the outer billiard table D, a semi-infinite cylinder, with respect 
to the vertical foliation consisting of tangent half-lines to the boundary of 
D. And finally, an n-periodic outer billiard orbit is an n-gon (possibly, star- 
shaped), circumscribed about the table D and having an extremal area. This 
is an outer analog of a variational property of periodic billiard trajectories: 
they are inscribed polygons of extremal perimeter length. As a consequence, 
if an outer billiard has a 1-parameter family of n-periodic orbits then the 
respective circumscribed n-gons have the same areas. 

Let Gn be the cyclic configuration space of the plane, that is, the space of 
polygons zi, . . . ,Zn such that Zi ^ Zi+i for i = 1, . . . , n (where the index i is 
understood cyclically). If 2;i(t), . . . , Zn{t) is a 1-parameter family of n-periodic 
outer billiard orbits with T{zi{t)) = Zi^i(t) then each segment Zi(t)zi+i{t) 
envelopes the boundary of the outer billiard table and is bisected by the 
tangency point. This prompts the following definition. 

Denote by JF the n-dimensional distribution in Gn determined by the 
following condition: the velocities of points zi, . . . , Zn lie in JF if, for each 



i = 1, . . . ,n, the motion of the hne -Zj-Zj+i is an infinitesimal rotation about 
the mid-point of the segment ZiZi^i. We call JF the dual Birkhoff distribution. 

The dual Birkhoff distribution is tangent to the level hypersurfaces of the 
area function, and we can restrict attention to non-degenerate polygons of 
a fixed non-zero area. On this hypersurface, J-' is non-integrable: the vector 
fields, tangent to JF, and their first commutators generate the tangent space 
at every point (see Theorem^. 

We are interested in the outer billiard tables possessing invariant curves 
consisting of n-periodic orbits. Such an invariant curve gives rise to a hor- 
izontal curve zi{t) , . . . , Zn(t) , < t < 1, in G^ satisfying the monodromy 
condition Zi{l) = Zi^i{0), i = 1, . . . ,n. If the outer billiard table is a circle 
(or an ellipse) then, for every pair (n, k) where 3 < n, 1 < fc < n/2, and 
k coprime with n, there is an invariant curve consisting of n-periodic orbits 
with rotation number k (that is, making k turns around D). We show that, 
for every such pair {n, k), one can perturb a circle so that the resulting outer 
billiard still has an invariant curve consisting of ri-periodic orbits with ro- 
tation number k; the space of such deformations is infinite-dimensional and 
depends on functional parameters (see Theorem |2)). 

In Section HI we study the simplest case of n = 3 in detail; this is the 
closest outer billiard analog of curves of constant width. This case illustrates 
the general approach by explicit computations. We also discuss a discretiza- 
tion of the problem in which a smooth outer billiard curve is replaced by a 
polygon. 

We also prove that, for every outer billiard table, the set of 3-periodic 
orbits has zero measure (see Theorem HJ. For inner billiards, this is a well- 
known result having a number of different proofs fn\ IT^ [T31 ITH] . We also 
give different proofs, one of them follows ideas of ^H] and another those of 
yj . Note that there exist area preserving twist maps of a cylinder whose set 
of periodic points has a non-empty interior. 

Let us finish this introduction with a problem: can invariant billiard 
curves (inner or outer) consisting of periodic points with different periods, 
or the same period but different rotation numbers, coexist, unless the table is 
an ellipse? In particular, is it true that the only billiard of constant width, 
having an invariant curve consisting of 3-periodic points, is a circle ? 



2 Configuration space of plane polygons and 
the dual Birkhoff distribution 

Let us fix some notations. Tlie determinant of two vectors, tliat is, tlie cross- 
product, is denoted by [ , ]. A generic point in tlie plane is denoted by 
z = {x,y). If zi = {xi,yi) and Z2 = {x2,y2) tfien Zidz2 denotes tlie 1-form 
Xidx2+yidy2- Likewise, [zi, dz2] denotes the 1-form Xidy2—yidx2 and [dz, dz] 
the 2-form 2dx A dy. li w = {u, v) is a vector in R^ and z = {x, y) is a point 
then by wdz we mean the tangent vector udx + vdy. When working with 
n-gons, we always understand the indices cyclically, so that i = i + n. For a 
polygon zi,...,2;„, set 

n 

A{zi, . . . jZn) = 2_J\.^i^ ^J+l]; 
1=1 

this is twice the area of the polygon (counted with appropriate multiplicities 
and signs). 

Let JF be the dual Birkhoff distribution on the space of n-gons defined in 
Section [TJ Let Z = {zi, . . . , 2;„) G (?„ be an n-gon and W = {wi, . . . , w„) a 
tangent vector to G„ at Z; the vector Wi is the velocity of the vertex Zi. 

Lemma 2.1 JF is an n- dimensional distribution. A vector W lies in T if 
and only if, for all i = 1, . . . ,n, one has: [wi + Wi+i, ZiJ^i — Zi] = 0, that is, 
the vectors Wi + Wj+i and Zj+i — Zi are collinear. 

Proof. To see that JF is ra-dimensional, view a polygon Z = (zi, . . . , Zn) G 
Gn as an ordered collection of n lines L = {ziZ2, Z2Z3, . . . , ZnZi). From this 
dual viewpoint, Gn becomes an open subset in the space £„ of n-tuples of 
oriented lines in the plane. An infinitesimal rotation of the i-th line ZiZi^i 
about the mid-point of the segment ZiZi+i determines a tangent vector to 
Cn at point L, and these vectors are clearly independent for i = 1, . . . , n. It 
follows that dim J^ = n. 

For W to belong to JF, the line through points Zi + ewi and Zi^i + ewj+i 
must pass through the mid-point {zi + 2;j+i)/2; here e is infinitesimal. Thus 
the linear term in e of the determinant 

h ewi+i, h ewi 



vanishes, which is equivalent to [wi + Wj+i, -Zj+i — Zi] = 0. □ 

An n-gon is called non- degenerate if no three consecutive vertices lie on 
one line. The set of non-degenerate n-gons is denoted by t/„. Clearly, [/„ is 
an open subset of Gn- 

Let us introduce the 1-forms a^ = [zj+i — Zi, dzi+i + dzi], i = 1, . . . ,n. 

Lemma 2.2 Let Z E Un be a non- degenerate polygon. Then the 1-forms Ui 
are linearly independent at Z and the intersection of their kernels is the fiber 
F{Z) of J-" at point Z. 

Proof. First, we claim that each form Oj vanishes on every tangent vector 
W E J-". Indeed, ai(W) = [zi+i — Zi,Wi + Wj+i], and this is zero, according 
to Lemma f2. 11 

Suppose that a non-trivial linear combination ^ tjOj vanishes at point Z. 
Then, for every test vector W = {wi, . . . , Wn), one has: 

= ^ ti[Zi+i - Zi,Wi + Wi+i] = ^ ti[Zi+i - Zi,Wi\+'^ ti[Zi+i - Zi, Wi+i] = 
^[ti{Zi+i-Zi),Wi]+^[ti-i{Zi-Zi^l),Wi] = y^}ti{Zi+i-Zi)+ti_i{Zi-Zi-i), Wj 

Therefore 

ti{zi+i- Zi) +ti-i{zi- Zi^i) = 0, i = l,...,n. (1) 

Since the polygon Z is non-degenerate, the vectors Zj+i — Zi and Zi — Zi^i are 
linearly independent, and (Hj) implies that tj = for all i. 

Since JF is n-dimensional and the 1-forms ai are linearly independent, one 
has: n^^i Ker ai = T. □ 

The next lemma shows that one can restrict attention to the level hyper- 
surfaces of the area function. 

Lemma 2.3 Every non-zero level hypersurface of the function A is smooth, 
and the distribution T is tangent to these hyper surf aces. 



Proof. Let Q : V -^ V* he a. self-adjoint linear operator in a vector space V. 
Consider the quadratic function F{x) = Q{x) ■ x on V^. Then every non-zero 
level hypersurface of F is smooth. Indeed, by Euler's formula, VF^x) ■ x = 
2F{x), hence dF does not vanish on a non-zero level hypersurface. The 
function A is a quadratic form on the space of n-gons, and the first claim 
follows. 

One has the equality 

n n 

^ «. = ^[2:^+1, dzi] - [zi, dzi+i] = -dA, 

i=l i=l 

which implies the second claim. □ 

Let us construct a system of n linearly independent vector fields tangent 
to JF. Let Z = {zi,...,Zn) be a non-degenerate polygon. Set a^ = [zi — 
Zj_i, Zj+i — Zi], twice the oriented area of the triangle Zj_iZj2;j+i. Denote by 
Wk = {wik, . . . , Wnk) the tangent vector to Un at Z whose i-th component is 
zero, unless i = k or i = k + l. For these values of i, one has: 

Wk,k = ak+l{Zk — Zk-l), Wk+l,k = Ctk{Zk+2 — Zk+l), 

see figure 0] 




Figure 3: Tangent vector Wk 

Lemma 2.4 The vectors Wi, . . . , Wn are linearly independent at every point 
Z E Un and they span F{Z). 
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Proof. To show that W^ hes in JF, it suffices to check that the vector 
(ik+iizk — Zk-i) +ak{zk+2 — Zk+i) is colhnear with the vector Zk^i — Zk- Indeed, 

[Zk+l — Zk, Zk+2 — Zk+l]{Zk — Zk-l) + [Zk — Zk-l, 2fc+l — Zk]{Zk+2 — ^fc+l) = 
[Zk — Zk-l, Zk+2 — Zk+l]{Zk+l — Zk), 

which follows from the general identity, valid for any three vectors: 

[u, v]w + [v, w]u + [w, u]v = 0. 

Under identification of the space of n-gons with the space of n-tuples of lines 
Cn, the tangent vector W^ corresponds to a nontrivial infinitesimal rotation 
of the line ZkZk+i about the mid-point of the segment ZkZk+i- Therefore these 
vectors are linearly independent and hence span the dual Birkhoff distribu- 
tion. □ 

Now we prove that the the dual Birkhoff distribution is completely non- 
integrable. 

Theorem 1 Let M^"-'^ be the hypersurface in Un consisting of non-degenerate 
n-gons with a fixed non-zero area. Then the tangent space of M at every point 
is generated by the vectors fields tangent to T and their first commutators. 
Thus the bracket growth type of T is (n, 2n — 1) {just like that of the Birkhoff 
distribution, see JT]). 

Proof. Consider the vector fields ^k = [Wk-i,Wk\ (the fields Wi and Wj 
clearly commute if |i — j| > 2). We want to show that these fields, along 
with JF, span the tangent space to M at every point. 

According to Lemmas 12.21 and 12.31 the 1-forms ai, . . . , a„_i constitute a 
frame in the conormal space to J-" at every point Z G M. For k = 1, . . . ,n, 
consider the vector (ai(^fc), . . . ,a„_i(^fc)) G R"^"*^. It suffices to show that 
these n vectors span R""^. 

Since the fields Wk belong to the kernel of each form ctj, the Cartan 
formula implies: aj(^fc) = dai(Wk-i, Wk). 

Recall that Z = {zi, . . . ,Zn) with Zi = {xi,yi). Let Ui = dxi A dyi, i = 
1, . . . , n. We claim that dai = 2(^01^+1 — Ui). Indeed, 

dtti = [dzi+i - dzi, dzi+i + dzi] = [dzi+i, dzi+i] - [dzi, dzi] = 2{uJi+i - Ui). 



Thus we want to show that the vectors 

((^2 - UJi){Wk-U Wk), ...,{uJn- UJn-l){Wk-l, Wk)) 

span R""^ for k = 1, . . . ,n. This will follow if we show that the vectors 

UJi{Wk-l, Wk),UJ2iWk-l, Wk), . . . ,^n(Wfc_i, Wk) 

span R" for fc = l,...,n. But uJi{Wk-i,Wk) = 0, unless i = k, and 
uJkiWk-i,Wk) = ak-iakttk+i 7^ 0. Thus uJi{Wk-i,Wk) is a non-degenerate 
diagonal matrix, and we are done. □ 

Remark 2.5 The dual Birkhoff distribution JF on the cyclic configuration 
space Gn is Lagrangian with respect to the following symplectic structure. 
The space of oriented lines in the plane, topologically, a cylinder, has a 
canonical area form (see, e.g., fSlE])! and the direct product of these forms 
is a symplectic structure in the space of n-tuples of oriented lines £„. In 
the proof of Lemma l2.H we realized Gn as an open subset in £„, and this 
provides a symplectic structure on Gn- Infinitesimal rotations of distinct 
lines determine symplectically orthogonal tangent vectors to £„, hence JF 
is Lagrangian. The Lagrangian property of the dual Birkhoff distribution 
echoes that of the Birkhoff distribution PP . 

3 Outer billiards with invariant curves con- 
sisting of periodic points 

In this section we show that, for every pair [n, k) with 3 < n,l < k < n/2 and 
k coprime with n, there is an abundance of outer billiard tables, possessing 
invariant curves consisting of n-periodic orbits with rotation number k; such 
tables are constructed as perturbations of circles. 

Denote by H^ the space of parameterized horizontal paths [0, 1] -^ M 
(paths tangent to the dual Birkhoff distribution) whose first 2 derivatives are 
square integrable (one can work with a higher number of derivatives as well). 
For a polygon Z G M, denote by H]r{Z) the space of paths starting at Z. 
Then H]r{Z) is a Hilbert manifold, see pUj . 

Let Z be a regular star-shaped n-gon with rotation number k. We have 
the following result. 
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Theorem 2 The set of (parameterized) outer billiard curves, sufficiently 
close to a circular one and possessing invariant curves consisting of n-periodic 
orbits with rotation number k, is a smooth Hilbert submanifold of codimension 
2n-l zn H]r{Z). 

Proof. To fix ideas, consider the case of simple periodic ra-gons, that is, the 
case of fc = 1. At the end of the proof, we will indicate the small adjustment 
to be made in the case oi k > 1. 

Start with a circular outer billiard. One has a 1-parameter family of 
periodic n-gons 

Z(t) = {z,{t),..., Zn{t)) with z,{t) = e2-(^-+*)/-, t E [0, 1] (2) 

circumscribed about the circular table (the table is scaled appropriately). 
Let (J : Gn — > Gn be the cyclic permutation of the vertices of a polygon: 
cr{zi, ...,Zn) = {z2, ...,Zn, Zi) . Then Z(l) = cr(Z(0)). 

The family Q determines a horizontal curve 7o(^) in Gn whose end- 
points are Z(0) and Z{1). Denote by H^{Z{0), Z{1)) C H^iZ{0)) the set 
of horizontal curves with the terminal point Z{1). We want to perturb 7o(t) 
in Hjr{Z{0),Z{l)). If a perturbation is sufficiently small and trivial in a 
neighborhood of end-points then we obtain a 1-parameter family of convex 
ra-gons whose vertices trace n arcs that smoothly match together to form an 
invariant curve of the outer billiard map consisting of ra-periodic points; the 
outer billiard curve is recovered as the envelope of the segments connecting 
the consecutive vertices. 

We want to show that Hjr{Z{0), Z{1)) is a Hilbert submanifold of codi- 
mension 2n - 1 in H^ZiO)). Denote by E : H]r{Z{0)) -^ M the end-point 
map that assigns the end-point 7(1) to a path 7. This map is smooth, and if 
E is a submersion at 70 G H]r{Z{0)) then i^|r(Z(0), Z{1)) = E-\Z{1)) is a 
codimension 2n — 1 submanifold, see PU] • A curve which is a singular point 
of the end-point map is called singular. 

To prove that 70 is not singular we use a criterion due to Hsu [7j . Denote 
by JF-*- c T*M the annihilator of the dual Birkhoff distribution and let Q be 
the restriction of the canonical symplectic form on the cotangent bundle to 
JF-*-. Let 7T : T*M — ^ M be the projection. A characteristic in JF-*- is a curve 
that does not intersect the zero section and whose direction at every point 
lies in the kernel of Q. By Hsu's criterion, a horizontal curve is singular if 
and only it is the projection under tt of a characteristic curve in JF-*-. 
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It is convenient for computations to use the following form of this crite- 
rion. The 1-forms a,, introduced in Section |2l generate T^: every covector 
in T^ can be written as ^ \iOLi where the coefficients Aj are "Lagrange mul- 
tipliers". There is exactly one relation: ^ctj = (see Lemma l2.3|) . and 
to factor this relation out, we assume that ^ Aj = 0. Consider the 2-form 
'^('^) = ^\(icxi on M. Then one has the following result, see (TU]: the 
kernel of fi at a point (x, ^ AjOj) G T^ is isomorphically projected by the 
differential dvr onto the kernel of the form uj[X) restricted to the fiber F[x) 
at point X G M. 

Let us find the 2-form C(j(A). From the proof of Theorem ^ we know that 
dai = 2{u!i^i — Ui) where cjj = dxi A dyi. Thus 

-2^('^) = 5Z ^'^^^ " ^^+i) ^ SZ^-^^ ~ \-\)^i- (3) 

i i 

Assume that 7o(t) lies in the kernel of the restriction of uj{X) on the fiber 
Filoit)): 

cu(A)(H^,.(7o(t)),7o(t))=0, k = l,...,n. (4) 

It is straightforward to compute that uJi(Wk{'yo(t)) , joit)) = unless i = k 
or i = A; -f 1; in these two cases, one has: 

MWk{lo{t)),Ut)) = C, Uk+i{Wk{^o{t)),Ut)) = -c 

where C is a non-zero constant depending on n only. Thus (jHl) and (^ imply 
that 2Aa; — Afc_i — \k+i = for all k. This, in turn, implies that all A^ are 
equal (indeed, by induction, A^+i = k\2 — {k — l)Ai for all fc, which implies, 
for k = n, that Ai = A2). Since ^ Aj = 0, all A^ vanish. It follows that the 
curve 7o is not singular. 

Finally, if the rotation number /c of a periodic trajectory is greater than 
1, one changes the curve © to 

Z(t) = (zi(t), . . . , Znit)) with z,it) = e2-(^-^+*)/", t G [0, 1], 

and the proof proceeds along the same lines. □ 



Remark 3.1 To illustrate the importance of checking that the curve 70 is 
not singular, let us give a simple example of a rigid horizontal curve, that is, 
a curve that does not admit smooth perturbations in the class of horizontal 
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curves with fixed end-points, see [TU]. Consider the curve 7(t) = (t, 0, 0), t G 
[0, 1] tangent to the distribution dz = y'^dx in 3-space. If {x(t),y(t),z(t)) is 
its perturbation as a horizontal curve with fixed end-points then 

= ^(1) - ^(0) = / y^{t)x'{t)dt. 
Jo 

Since x'(t) > 0, we have y(t) = 0, and hence z(t) = 0, for all t; thus the 
perturbed curve is a reparameterization of 7. 

4 Case study: n = 3 and discretization 

Three is the smallest possible period of the outer billiard map. In this section 

we illustrate the general constructions of Sections |21 and El in the case n = 3. 

Let 7 be a smooth strictly convex outer billiard curve and Z1Z2Z3 a 3- 

periodic orbit. Let wi, 102,103 be the mid-points of the sides of the triangle 

Lemma 4.1 The inscribed triangle wiW2Ws has the property that, for every 
i = 1,2, 3, the tangent line to 7 at vertex Wi is parallel to the side Wi+iWi-^2- 
Conversely, if an inscribed triangle has this property then the tangent lines 
to 7 at its vertices form a circumscribed triangle whose sides are bisected by 
the tangency points. Furthermore, this property of an inscribed triangle is 
equivalent to having extremal area. 

Proof. The first claim holds by definition of the outer billiard map and the 
second follows from elementary geometry. To prove the last statement, fix 
points W2 and w^ and vary Wi. The area is extremal when the distance from 
Wi to the line ^2^3 has an extremum, and this happens if and only if the 
tangent line to 7 at wi is parallel to W2W3. □ 

It follows that we can replace circumscribed triangles of extremal areas 
by inscribed ones. We want to construct smooth strictly convex closed curves 
such that an inscribed triangle of extremal area can be turned around inside 
the curve. 

Following the approach of Section |2l let M be the 5-dimensional manifold 
of triangles of area 3/2. Let W1W2W3 be a triangle from M; denote by c 
its center of mass. Let u = wi ~ c,v = W2 — c; then [u, v] = 1 and the 
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frame {u,v) belongs to 5'L(2,R). The cyclic group Z3 acts on M by the 
cyclic permutations of the vertices of a triangle and on SL{2, R) by the 
transformation T{u, v) = {v, —u — v). Extend the action of Z3 to SL{2, R) x 
R^ taking product with the trivial action on the second factor. On M , we 
have a 3-dimensional distribution J-' determined by the condition that the 
velocity of Wi is parallel to the line Wi+iWi+2, 'i = ^,'^, 3. 

Lemma 4.2 The correspondence {wi, W2, w^) 1— > {{u, v), c) is a Z^-equivariant 
diff'eomorphism M -^ SL(2, R) x R^. The fibers of the projection SL{2, R) x 
R^ — > SL{2, R) are transverse to the distribution T . 

Proof. The first claim is obvious. To prove the second, note that a vector, 
tangent to the fibers of the projection S'L(2, R) x R^ — ;> SL{2, R), is induced 
by a parallel translation of the triangle: all three vertices move with the same 
velocity. If this vector lies in JF then it must be parallel to all three sides of 
the triangle, which is impossible. □ 

We want to construct a closed horizontal curve 7(t), t G [0, 27r] in M 
satisfying the monodromy condition 

7(t + 27r/3)=a(7(t)) (5) 

where a is the cyclic permutation of the vertices of a triangle. By Lemma 1^^ 
this curve projects to a smooth curve 7(t) in SL{2, R) satisfying 7(t+27r/3) = 
T(7(t)). Conversely, such a curve 7(t) C 5'L(2,R) lifts uniquely to a hori- 
zontal curve 7(t) C M, provided a lift of the initial point is chosen, but the 
monodromy condition (0) may fail. But once (0) is satisfied - and assuming 
convexity - we obtain a desired plane curve possessing a 1-parameter family 
of inscribed triangles of extremal areas. 

Let us work out explicit formulas. Denote by 7(t) = {u{t),v(t)), t e 
[0, 27r] a closed curve in S'L(2,R) such that u{t + 27r/3) = v{t) and v{t + 
27r/3) = —u{t) — v{t). Let c{t) be the curve of the center of mass of the 
respective triangle, that is, the projection to R^ of a lifted horizontal curve 
7(t). Set: 

[u{t),u\t)]=p{t), [v{t),v\t)]=q{t), [u'{t),v{t)] = [v'{t)Mt)]=r{t). 
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Proposition 4.3 The velocity of the center of mass is given by the formula: 



c{t) = ^{p{t) - 2q{t) + 2r(t)) v{t) - ^(g(t) - 2p{t) + 2r(t)) u{t). (6) 
The monodromy condition ^ holds if and only if 

(p(t) - 2q{t) + 2r(t)) v{t) - {q{t) - 2p{t) + 2r{t)) u{t) dt = (7) 





or, equivalently, 



\%i{t),v{t)\{v{t)-u{t)) dt = 0. (8) 





Proof. One has: 

Wi{t) = u{t) + c(t), W2it) = v{t) + c{t), wsit) = -u{t) - v{t) + c{t). (9) 
The curve is horizontal if and only if 

[W2{t)-w^{t),w'^{t)] = K(t) - W2(t),K(i)] = K(t) -^3(t),^(i)] = 

or, in view of (jH)), 

[u{t),(^{t)] + 2[v{t),d{t)]=2r{t)-p{t),2[u{t),d{t)] + [v{t),d{t)]=2r{t)-q{t) 

(there are only two equations because the third is their consequence). It 
follows that 

3[n(t),c'(t)] = 2r(t) - 2p{t) + g(t), ?>[v{t),d{t)] = 2r{t) - 2q{t)+p{t), 

and since [u{t),v{t)] = 1, this implies ©• 

Abusing notation, let T be the shift t ^— > t + 27r/3 of the argument of the 
functions u{t) ,v(t) , p(t) , q(t) , r(t); this defines an action of Z3 on the space 
generated by these functions. One has: 

T : u ^-^ V ^—>- —u — V, p \-^ q \—>- p + q — 2r, r \-->- — r + q \-^ p — r. (10) 

The monodromy condition Q is satisfied if and only if c(t) is 27r/3-periodic. 
Applying (fTn|) to (jHl), it follows that c'(t) is 27r/3-periodic, and hence c(t) is 
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27r/3-periodic if and only if c{t) is 27r-periodic, that is, if j^^ c'{t)dt = 0. In 
view of (jni, this is equation (jZj). Finally, for every function f{t), one has 

\{f){t)dt= r f{t)dt. 

Jo 

Applying this to (fTn|) yields four identities for integrals of vector-functions 
pu,pv, qu, qv,ru,rv: 

2tt /■27r /■27r /•2tt 

q{t)u{t) dt = - r{t)v{t) dt, / p{t)v{t) dt = - r{t)u{t) dt, 

Jo Jo Jo 

'•271" /'27r /'27r /'27r 

q{t)v{t)dt= I p{t)u{t)dt= / r{t)u{t)dt+ / r{t)v{t) dt. 
'0 Jo Jo Jo 

These identities imply that ((7j) and (jH)) are equivalent. □ 

To summarize, one constructs an outer billiard table having a 1-parameter 
family of 3-periodic trajectories from a closed curve in SL{2, R) satisfying 
(jHI), that is, satisfying two scalar equations. 

It is interesting to consider the following discretization of the problem in 
which a smooth curve is replaced by a polygon. Let P be a convex polygon 
and ABC an inscribed triangle whose vertices are some vertices of P. One is 
allowed to shde one vertex of the triangle along an edge of P to the adjacent 
vertex of P, provided this edge is parallel to the opposite side of the triangle. 
Using a sequence of such moves, one wants to turn the triangle around inside 
P. For which polygons is this possible? 

Consider an example. Let P be a regular k = 3n + 1-gon with vertices 
zi, . . . Zk, and let A = zi,B = Zn+2,C = Z2n+2- Then the segment Z1Z2 is 
parallel to the segment BC, and one can move point A to Z2- After that 
one can move point C to Z2n+3, then point B to Zn+3, etc. - see figure El for 
k = 7. 

One can deform a regular k = 3n + 1-gon preserving the desired property: 
the condition to satisfy is that the segments -Zj-Zj+i and Zi+n+iZi+2n+i be 
parallel for all i = 1, . . . ,k. This gives k conditions, whereas a k-gon has 2k 
degrees of freedom, so it is not surprising that non-trivial deformations exist. 
We do not dwell on details here. 

Let us finish this section with a problem. Suppose that a convex domain 
has the property that an inscribed triangle of extremal area can be turned 
around inside the domain. Find the lower and upper bounds for the ratio of 
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Figure 4: Rotating a triangle inside a regular septagon 

the areas of the triangle and the domain. We conjecture that a parallelogram 
and an ellipse are two extreme cases. 

5 The set of 3-periodic orbits has empty in- 
terior 

It is an old conjecture, with motivation in spectral geometry, that the set 
of n-periodic billiard trajectories in a billiard with a smooth boundary (not 
necessarily convex) has zero measure. For n = 2, this is easy to show, and 
for n = 3, this is a theorem of Rychlik |E1 with a number of different proofs 
available [TJ [T21 US HEl ■ For n > 4, the conjecture is open. 

This section concerns the outer counterpart of this conjecture. Consider 
n outer billiard tables Di, . . . , D„ with smooth boundaries and let Ti, . . . , T„ 
be the respective outer billiard maps. We are interested in the set F of fixed 
points of the composition T„ o T„_i o ■ • ■ o Ti. 

Conjecture 3 For every n > 3, the set F has zero measure. 

Remark 5.1 For strictly convex outer billiards with corners. Conjecture |H1 
fails. An example shown in figureElis a rounded square: the set of 4-periodic 
orbits has a non-empty interior. One can generalize: if a plane polygonal 
outer billiard has an n-periodic orbit then a whole neighborhood of this point 
consist of periodic points as well (with period 2n, if n is odd). According to 
a theorem of Culter [2], every polygonal outer billiard has periodic orbits,^ 
thus one can use any (rounded) polygon to construct an open set of periodic 
orbits. 

We prove the following particular case of Conjecture El 



*For inner polygonal billiards, this is an outstanding open conjecture. 
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Figure 5: An open set of 4-periodic orbits for a piece-wise smooth outer 
billiard table 

Theorem 4 Let Di,D2, -D3 be three smooth strictly convex outer billiard ta- 
bles and Ti,T2,T3 the respective outer billiard maps. Then the set F of fixed 
points of the map T3 o T2 o Ti has empty interior. 

As in im IT^ [T^ ll6j. this theorem implies that F has zero measure; we 
do not dwell on this refinement. 

As a preparation to the proof of Theorem EJ let us recall two results 
on outer billiards. The first is an outer analog of the "mirror equation" 
of geometrical optics that plays an important role in the theory of inner 
billiards. 

Let D be an outer billiard table with smooth boundary and T the outer 
billiard map. Let T(x) = y, see figure El 




Figure 6: The differential of the outer billiard map and the mirror equation 
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Proposition 5.2 In the Cartesian coordinate system in figure\^ the differ- 
ential of T at point x is given by the matrix 

/ _1 _2£ 
dT = i '■ 

where r = \xy\/2 and p is the radius of curvature of the outer billiard curve 
at the reflection point. If dT takes a line I to a line m, see figure{^ then one 
has the following "mirror equation": 

cot a + cot /5 = — . (11) 

r 

For a proof of Proposition 15.21 see jB] . 

Ttie second result is tlie following "area construction" , an outer counter- 
part of the string construction in the theory of inner billiards, that recovers 
a billiard table from its caustic. 

Proposition 5.3 Let D be an outer billiard table and C an invariant curve 
of the outer billiard map T . Let a; G C and T{x) = y. Then the area cut by 
the segment xy from C is the same for all x E C . In other words, D can be 
recovered from C as an envelope of segments of constant areas. 

For a proof of Proposition 15. 3[ see JHIIISIIIII or [H]. 

In particular, let C consist of two non-parallel lines, / and m, as in figurelHl 
Then the area construction yields a hyperbola for which the lines / and m are 
the asymptotes. If the outer billiard map T takes xtoy and dT{l) = m, as in 
figure El then the hyperbola for which the lines / and m are the asymptotes 
is second order tangent to the outer billiard curve at the mid-point of the 
segment xy. Note also that of 4 possible orientations of the pair of lines I 
and m in figure El only two are consistent with the action of dT: one line is 
oriented toward their intersection point while the other is oriented from it. 

Proof of Theorem |31 We give two proofs. The first is reminiscent of the 
one given in [16j. 

Consider figured One has three outer billiard maps, Ti, T2, T3 that cycli- 
cally permute points Zi,Z2,Z3: 

Tl{z2) = Z3, T2{zs) = Zi, Tsi^Zi) = Z2. 
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-i '2 

Figure 7: Proving Theorem 0] 



Let Oj, i = 1,2,3, be the angles of the triangle, 2rj its side lengths and pi 
the curvature radii of the outer billiard curves at the reflection points. Set 
dTi = Ai and let Bi be matrix of rotation through angle vr — Oj. 

Assume that the orbit zi, Z2, z^ belongs to an open set U of fixed points 
of the map T3 o T2 o Ti. Then d{T3 0T2 oTi) is the identity in U . Using 
Proposition 15.21 this implies that 



Si^^aAi^sAa = Id or A^B2A^ = B-^ A:,^ B'^ 
The matrices involved are as follows: 



(12) 



A 



-1 



Bi 



- cos Oj 

sin «,■ 



• sm ttj 
cos a,- 



It is straightforward to compute the triple products of the matrices on both 
sides of the second equation (fT^ . Apply both products to the vector (1,0) 
and equate the second components: 



sma;2 
and since 02 = vr — ai — 0:3, 



— sin(Q;i + as) + 2sina;i sin as — , 

r2 



P2 
^2 



sin(ai + as) 
sinai sin as 



cot ai + cot as. 



(13) 
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Let A be the area of the triangle zi,Z2,Z3 and /i2 its altitude from the 
vertex Z2. Then by elementary geometry, 

2ri sin a^ = 2r^ sin ai = /i2, ri cos 0:3 + r^, cos ai = r2, A = /i2^2, 

which makes it possible to rewrite (fT!^ as 

Ap2 = 24. (14) 

For orbits 21,22, -23 in the set ?7, the area A remains constant. Move points 
zi and Zs slightly along the line ziz^ so that the mid-point of the segment 
212:3 remain the same. We obtain a family of orbits in U for which p2 does 
not change but r2 does, in contradiction with (fT^ . This completes our first 
proof. 

The second argument is a variation on the proof given in jl]. Let M^ be 
the manifold of non-degenerate triangles whose oriented areas equal 1 and 
let T^ be the dual Birkhoff distribution on M. If there is an open set of 
fixed points of the map T3 o T2 o Ti then one has a 2-dimensional disc U <Z M 
tangent to T. We will show that this is impossible. 

If such a disc U exists, let x and y be local coordinates in it. Then 
u = d/dx and v = d/dy are commuting vector fields on U. 

As a basis of vector fields in M generating JF we can choose the fields 

Wi = (22 - Zi)dz2 + (21 - Z3)dz3, W2 = (23 - ^2)^-23 + (-22 " 2i)92i 

and 

W3 = (21 - ^3)^-21 + (-23 - 22)5^2, 

see Sectional Then 

u = fiWi + f2W2 + fsWs, V = g^Wi + g2W2 + gsW^ 

where fi,gi are smooth functions on U. Without loss of generality, /i 7^ 
and (72 7^ 0. Then we take a linear combination of u and v, with coefficients 
that are smooth functions, to obtain new fields 

u = W, + fWs, v = W2 + gW3, 
so that [u,v] G Span('U,'i;). In particular, [u,v] is tangent to JF. 
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It follows that daJu.v) 



for i = 1,2,3 where a,- are the 1-forms 



introduced in Section |21 It is straightforward to compute that 

da2{u, v) = -2(1 + g), das{u, v) = 2(1 + /), 

and hence f = g = -1. Thus u = Wi - W3, v = W2 - W3. 
Now one computes the commutator: 

[Wi - W3, W2 - W^] = [Wi, W2] + [1^2, W^] + [PFs, Wi] = 4{Wi + W2 + W3). 

Since the fields Wi,W2,W3 are everywhere linearly independent, the vector 
VFi + H/2 + W3 is not a linear combination of Wi — W3 and W2 — W3. This 
is a contradiction proving that the disc U does not exist. □ 

Remark 5.4 The reason we consider n outer billiards in Conjecture El in- 
stead of just one, is that this makes it possible to relax the convexity assump- 
tions and include outer billiard tables, convex outward (similarly to the case 
of inner billiards where one does not assume the billiard table to be convex). 
In fact, the proof of Theorem |3] considerably simplifies if one considers 
a single strictly convex outer billiard table. Namely, let zi be a 3-periodic 
point of an outer billiard map T. Then the differential dT^{zi) cannot be 
equal to identity. 





Zl 




Z2 



Figure 8: Can dT^ = Id? 

Indeed, let T{zi) = Z2,T{z2) = z^jT^z^) = Zi, see figure |H1 Assume that 
dT^{zi) = Id. Let u and v be the opposite unit tangent vectors at points 
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Zi and Z2 having the direction of the side -21-22- Then dT{u) = v. Let w = 
dT{v) = dT~^{u). Since T is an orientation preserving transformation, the 
vector w hes on the same sides of the hnes 2:2^3 and Z1Z3 as the outer bilhard 
table D. Thus w hes inside the wedge Z1Z3Z2. Let A be the intersection point 
of the hne through point Z3 in the direction of vector w with the hne ziZ2- 
It follows from the area construction (see the paragraph after Proposition 
15 .31) that the outer billiard curve is second order tangent to the hyperbola 
inscribed in the angle ziAzs and tangent to the segment Z1Z3 in its mid-point. 
This hyperbola is convex outward, contradicting the convexity of D. 

On the other hand, d(T2 o Ti o T3) may be equal to the identity at a fixed 
point of the map T2 o Ti o T3. An example is shown in figure |S1 on the right: 
Zi-22-23 is a unit equilateral triangle, and the three outer billiard tables are 
circles of radii l/^/S. Then equation (fT^ holds, and d{T2 o Ti o T3)(,2i) = Id. 
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